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1.0  Introduction 


The  main  objective  of  this  project  is  to  assess  the  feasibility  of 
developing  a  mathematical  model  for  IR  imaging  sensors  which  could  be 
used  in  real-time  stereo  compilation  equipment  and  tactical  Air  Force 
targeting  app 1 i ca t i ons .  The  work  was  divided  as  follows. 

3.  A  survey  of  existing  IR  sensors  and  imaging  systems. 

2.  A  survey  of  photogrammetric  techniques  and  procedures  for 
stereo  viewing  and  plotting. 

3.  Development  of  a  flexible  system  model  which  can  be  adapted  to 
various  types  of  IR  imaging  systems. 

During  this  work,  particular  attention  was  paid  to  integrating  IR 
imagery  into  existing  equipment  and  procedures.  Recommendations  are 
also  made  with  regard  to  future  systems  which  will  be  able  to 
capitalize  on  emerging  computer  graphics  technologies. 

The  organization  of  the  report  is  as  follows.  Section  2  gives  a 
brief  overview  of  IR  imaging  and  the  available  types  of  IR  detectors 
and  focal  plane  arrays.  This  overview  has  been  limited  to  the 
technologies  most  likely  to  be  used  for  modern  IR  imaging  systems. 
Section  3  discusses  photogrammetric  considerations  and  furthermore 
concentrates  on  those  most  pertinent  to  the  needs  of  Air  Force  and  DHA. 
Section  A  introduces  and  describes  the  system  model  which  was  developed 
and  is  adaptable  to  any  one  of  the  candidate  imaging  systems.  Finally, 
Section  5  presents  the  conclusions  of  this  work  and  discusses  various 
alternatives  in  integrating  IR  imagery  into  existing  operations. 
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2.0  Infrared  Imaging 


There  are  several  kinds  of  IR  imaging  systems.  This  section 
briefly  describes  various  IR  imaging  systems  and  concentrates  and  those 
most  likely  to  be  used  for  aerial  imagery  collection. 

2.1  Comparison  Between  IR  and  Visible  Imagery 

Conventional  visible  photography  is  a  very  mature  technology. 
Over  the  years,  improvements  in  photographic  emulsions  have  yielded 
films  which  are  capable  of  very  high  resolution.  Visible  photography 
lies  roughly  in  the  .4  to  .7  micron  region  of  the  electromagnetic 
spectrum.  Typically,  aerial  photographic  systems  filter  out  light 
toward  the  blue  end  of  the  spectrum. 

Photographic  films  are  ivailable  which  are  sensitive  into  the  near 
infrared  (.7  to  .9  micron).  Thes-  films  are  used  to  image  IR  radiation 
reflected  from  objects  as  opposed  to  IR  radiation  radiated  from 
objects.  Usually  the  source  of  IR  radiation  is  the  sun,  however,  for 
some  applications  an  artifical  IR  source  is  used  to  illuminate  the 
scene. 

The  primary  reason  IR  photography  and  electronic  imagery  lag  their 
visible  counterparts  is  the  lack  of  photoconductive  and  photoemissive 
materials  which  operate  at  IR  wavelengths.  Both  photoconductivity  and 
photoemissivity  are  quantum  processes  which  depend  upon  the  absorption 
of  photons.  Photons  with  wavelengths  greater  than  approximately  i 


micron  simply  do  not  have  sufficient  energy  to  be  absorbed  by  most 
photoconductive  and  photoemmissive  materials.  There  are  some  exotic 
materials  which  work  further  out  into  the  infrared,  however,  they  have 
other  limitations  such  as  the  need  to  be  cooled  and  low  sensitivity. 
In  recent  years,  significant  advances  in  electronic  IR  imaging  devices 
have  been  made.  The  most  popular  of  these  are  Charge-Coupled-Device 
(CCD)  and  Charge-Injection-Device  (CID)  unit  cell  focal  plane  arrays. 
Linear  or  rectangular  arrays  of  these  unit  cells  are  often  operated  in 
a  time-delay-and-integrate  (TDI)  mode  in  order  to  increase  the 
sensitivity  of  the  array.  In  this  mode  both  the  scene  and  its 
corresponding  electrical  analog  move  along  the  array  in  a  "pushbroom" 
fashion.  This  type  of  operation  actually  forms  an  electronic 
equivalent  of  a  strip  camera.  A  brief  description  of  strip  cameras 
will  be  given  in  a  later  section. 


2.2  Detector  Cells 

The  unit  detector  cell  is  the  most  elementary  part  of  the  detector 
array.  The  unit  cell  which  is  chosen  governs  the  application  for  which 
the  array  shall  be  used  in  addition  to  determining  the  overall 
performance  and  operation  of  a  particular  array.  The  primary  function 
of  the  unit  cell  is  to  convert  IR  radiation  to  a  usable  electrical  form 
which  then  undergoes  various  stages  of  signal  processing.  Requirements 
for  unit  cell  operation  include:  low  noise,  low  circuit  complexity, 
high  uniformity  of  response,  high  injection  efficiency,  and  high 
dynamic  range  [l] .  The  most  popular  types  of  unit  cells  are  those 
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which  employ  charge  coupling  (CCD)  and  charge  injection  (CID) .  Neither 
device  is  better  than  the  other,  however,  they  each  have  their  own 
advantages  and  disadvantages. 

2.2.1  CCD  Unit  Cells 

The  CCD  is  probably  the  more  popular  of  these  two  unit  cells 
mainly  due  to  its  simplicity  of  operation  and  fabrication.  The  basic 
CCD  unit  cell  is  shown  in  Figure  1. 

IR  radiation  strikes  the  detector  portion  of  the  unit  cell, 
characterized  by  a  photodiode,  which  is  coupled  into  an  integrating 
well.  This  well  is  basically  a  capacitor  which  stores  accumulated 
charge.  A  transfer  gate  provides  the  path  for  current  to  flow  from  the 
photodiode  to  the  well.  A  certain  amount  of  time,  called  the 
integration  time,  is  required  to  fill  the  well.  Each  well  is  then 
electronically  scanned  by  a  shift  register  which  transfers  the  charge 
to  the  next  CCD  cell.  Eventually  the  charge  is  transferred  out  of  the 
array  and  the  process  can  be  repeated. 

The  charge  capacity  of  the  CCD  is  dependent  upon  several  factors. 
The  maximum  charge  storage  possible  is  limited  by  the  density  of  the 
array  as  this  limits  the  size  of  the  well.  The  maximum  voltage  which 
may  be  applied  to  the  photodiode  is  limited  by  breakdown  and  tunneling 
considerations.  In  addition,  overcharging  of  the  well  must  be  avoided 
otherwise  an  effect  known  as  blooming  can  occur.  Lastly,  there  is  a 
strong  temperature  dependence  for  the  charging  mechanism.  Typically 
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these  detectors  are  kept  at  a  constant  temperature  (often  77°K)  to 
limit  the  amount  of  dark  current.  Dark  current  flows  when  the 
photodiode  is  not  illuminated  and  ideally  no  current  is  flowing. 
Performance  of  the  CCD  cells  is  also  affected  by  the  purity  of 
materials  used  in  the  fabrication  of  the  well  and  detector  [2,3]. 

Another  consideration  for  CCD  operation  is  the  accessibility  of 
the  electrical  signals  from  each  CCD  cell.  Figure  2  provides  an 
illustration  of  two  schemes.  One  possible  way  to  transfer  signals  out 
is  serially  via  a  multiplexer.  This  method  is  used  for  applications 
where  long  intervals  are  required  due  to  excessive  signal  processing. 
Another  method  interfaces  in  a  parallel  fashion  and  thus  allows  for 
shorter  integration  times.  Since  integration  times  are  basically 
controlled  by  the  clock  rate  of  the  shift  registers,  these  rates  may  be 
varied  electronically  to  provide  variable  sensitivity. 

2.2.2  CID  Unit  Cells 

The  CID  detector  may  be  described  as  a  surface  channel  device  that 
employs  intracell  charge  transfer  and  charge  injection  to  achieve  the 
solid  state  image  sensing  function  [A].  A  basic  CID  cell  is  shown  in 
Figure  3.  This  cell  basically  consists  of  a  detector,  a  metal-oxide 
semiconductor  (MOS)  capacitor,  and  gates.  Photocu-rent  is  generated 
when  IR  radiation  falls  on  the  detector  which  in  turn  charges  the  MOS 
capacitor.  As  the  capacitor  charges  it  is  basically  integrating.  At 
some  point  in  the  charging  period,  a  gate  is  closed  connecting  the 


capacitor  to  the  substrate,  discharging  charging  and  discharging  is 
controlled  by  an  electrical  network  as  illustrated  in  Figure  4.  This 
electrical  network  is  called  an  x-y  matrix  which  allows  any  cell  to  be 
addressed  and  thus  read.  There  are  various  sensing  schemes  used  which 
include  sequential  injection,  preinjection,  and  parallel  injection. 
Sequential  injection  scans  the  matrix  by  selecting  a  row  and  pulsing 
the  sequential  columns  to  read  the  row.  Reading  all  rows  sequentially 
results  in  a  raster  scan  of  the  detector  array.  The  preinjection 
technique  relies  on  sensing  a  change  in  charge  when  a  complete  row  is 
injected  simultaneously.  Each  row  is  charged  (preinjected)  prior  to 
sensing  and  then  each  detector  cell  discharges  proportional  to  its 
photocurrent.  The  change  in  charge  is  measured  when  the  charge  of  a 
detector  is  injected  into  the  substrate.  Parallel  injection  allows  the 
function  of  charge  detection  to  be  separated  from  charge  injection. 
The  rows  are  selected  and  sensed  in  parallel  allowing  fast  detection. 
The  performance  of  CID  arrays  is  dependent  on  several  factors.  First 
of  all,  temperature  plays  a  large  part  in  the  production  of  dark 
current  which  directly  effects  the  dynamic  range.  The  maximum  voltage 
is  determined  by  the  breakdown  and  tunneling  of  MOS  devices  used  as 
gates.  Lastly,  high  data  rates  and  low  signal  levels  of  the  cells 
require  the  use  of  processing  amplifiers  that  are  low  in  noise  and  high 
in  bandwidth.  This  additional  circuitry  must  be  small  in  size  and  low 
in  power  consumption  [5] . 
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2.3  Operational  Array  Modes 


Depending  upon  system  resolution,  cost,  and  application,  several 
operational  array  modes  may  be  chosen.  These  modes  may  be  divided  into 
two  basic  groups,  staring  arrays  and  scanning  arrays.  The  difference 
between  these  two  modes  is  that  staring  arrays  continuously  "look"  at 
the  scene  while  scanning  arrays  are  time  gated. 


2.3.1  Staring  Arrays 

Staring  arrays  can  be  constructed  as  either  two-dimensional  mosaic 
arrays  or  as  one  dimensional  linear  arrays.  A  typical  staring  array 
system  is  pictured  in  Figure  5.  In  this  diagram,  energy  emitted  from 
the  scene  is  focused  onto  a  two-dimensional  mosaic  array  of  IR 
detectors.  The  mosaic  array  is  typically  constructed  from  a  large 
number  of  detectors  arranged  in  a  two  dimensional  pattern.  One  method 
of  fabricating  such  an  array  is  to  butt  several  smaller  arrays  say,  for 
example,  1024  x  64  together.  This  is  often  necessary  because  of  the 
difficulty  of  constructing  large  mosaic  arrays  in  one  fabrication. 

One  disadvantage  of  staring  arrays,  however,  is  their  lack  of 
sensitivity.  One  commonly  used  way  to  increase  the  sensitivity  is  to 
operate  detectors  in  a  mode  which  allows  the  sensors  to  integrate 
signal  over  as  long  a  time  as  possible.  A  method  of  achieving  this 
integration  of  the  signal  is  called  Time-Delay-and-Integrate  (TDI) . 
There  is  a  practical  limit,  however,  on  the  length  of  integration  time. 
The  detectors  basically  accumulate  charge  during  the  integration  time 
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and,  tine*  each  detector  alaaant  can  hold  ao  much  charge,  tha  amount  of 
aecuaulatad  charga  i>  limited. 

Thara  ara  othar  problaas  associated  with  aoaaic  ataring  arraya. 
Most  of  thaaa  problaas  originate  from  tha  single  aost  critical  part  of 
the  systaa,  tha  aoaaic  array.  Because  a  large  number  of  detectors  ara 
used  to  make  up  a  focal  plana,  non-uniformity  of  the  responsivity  of 
each  detector  may  present  a  problem.  In  addition,  the  assembly  of  a 
large  focal  plane  by  butting  smaller  arrays  can  cause  a  loss  of  active 
elements  along  a  butt  seam  resulting  in  lower  resolution  along  seams. 
Lastly,  when  large  integration  times  are  used,  large  background  levels 
can  swamp  out  the  smaller  AC  signal  [2] . 


2.3.2  Scanned  Arrays 

An  alternative  to  staring  arrays  is  the  scanned  array  [3~5] . 
Radiation  emitted  from  the  scene  is  reflected  onto  the  detector  array 
by  a  rotating  mirror  or  prism.  When  the  orientation  of  the  mirror  is 
such  that  it  reflects  the  scene  radiation  normal  to  the  detector  array, 
the  detectors  are  illuminated.  By  varying  the  speed  of  rotation  of  the 
mirror,  the  amount  of  radiation  from  the  scene  incident  on  the  detector 
can  be  controlled.  The  scanning  mirror,  therefore,  can  provide  the 
same  function  as  the  shutter  in  a  camera.  The  orientation  of  the 
linear  array  with  respect  to  the  projected  scene  is  vital.  The  linear 
array,  which  is  rectangular  in  shape,  is  orientated  such  that  the 
direction  of  motion  of  the  array  it  perpendicular  to  the  long  side  of 
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the  array  as  illustrated  in  Figure  6.  By  having  such  an  orientation,  a 
rectangular  strip  iaage  aay  be  recordej  every  tiae  the  airror  projects 
radiation  onto  the  array.  The  detector  iaages  aay  be  electronically 
stored  and  asseabled  to  fora  a  single  continuous  iaage  of  width  w  and 
length  nL  where  n  is  the  nuaber  of  rotations  of  the  airror. 

2. A  Construction  of  Focal  Plane  Arrays 

Depending  upon  cut-off  wavelength,  complexity,  and  cost,  several 
construction  and  fabrication  techniques  aay  be  chosen  for  the 
aanufacture  of  focal  plane  arrays.  There  are  basically  two  classes  in 
which  these  arrays  aay  be  divided,  monolithic  and  hybrid.  In  general, 
aonolithic  focal  planes  collect,  transfer  and  store  charge  from  IR 
radiation  in  a  single  medium.  Hybrid  focal  planes,  however,  detect  the 
IR  radiation  in  a  sensing  medium  which  is  electrically  and  mechanically 
connected  to  a  different  medium.  The  materials  composing  the  detectors 
and  IR  semiconductors  include  PbTe,  PbSnTe,  InSb,  InAsSb,  InGaSb,  PbS, 
PbSeTe ,  and  HgCdTe  [5]. 

Desirable  properties  of  the  focal  plane  array  include:  high 
uniformity  of  response,  high  flexibility  for  tests  and  diagnosis,  and 
low  dead  space  [l] .  The  first  and  third  of  these  depend  on  mechanical 
design  and  construction  of  the  array.  Non-uniform  response  may  also  be 
caused  by  non-uniform  temperature  distributions  or  impurities  in  the 
substrate.  Excessive  dead  space  may  be  caused  by  the  mechanical 
imprecision  of  chip  butting  which  is  necessary  in  the  assembly  of 


mosaic  arrays.  Testing  and  diagnosis  may  include  a  self  test  to 
determine  whether  each  cell  or  array  is  operating  properly.  These 
tests  are  only  possible  with  additional  hardware  and  thus  higher 
electronic  complexity. 

Depending  on  the  materials  used  for  the  array  and  detector 
manufacture,  different  cut-off  wavelengths  are  attainable.  However, 
each  of  these  materials  has  advantages  and  disadvantages  which  must  be 
considered.  As  to  actual  manufacture  of  focal  planes,  the  monolithic 
technology  is  probably  the  most  limited.  One  limitation  is  in  the 
detector  region  since  each  cell  cannot  be  individually  optimized.  The 
reason  for  this  lies  in  the  fact  that  the  optimization  of  one  layer  at 
an  elevated  temperature  will  lead  to  the  degradation  of  the  other  [8]. 
This  temperature  problem  is  inherent  since  focal  plane  temperatures  are 
often  kept  at  77 °  K.  The  main  advantage  of  the  monolithic  technology, 
however,  is  its  relatively  low  production  cost  due  to  its  simplicity  of 
const  rue  t i on . 

The  hybrid  focal  plane  array  may  be  described  as  an  integrated 
mul t i ~e 1 ement  array.  Therefore,  increased  mechanical  and  thus  higher 
construction  complexity  result.  There  are  several  methods  which  are 
used  to  mechanically  mount  the  detectors.  Examples  of  two  common 
methods  are  direct  and  flip  chip  or  solder  bump  mounting  [5], 

An  additional  consideration  in  array  manufacture  is  electrical 
connection  of  detectors.  In  mosaic  arrays  particularly,  there  is 
usually  some  scheme  for  reading  the  individual  detectors  (e.g.  serial 


or  parallel).  Therefore,  the  necessary  reading  lines  could  contribute, 
to  a  large  extent,  to  the  complexity  of  each  array. 

2.5  Practical  Considerations 

In  any  system  which  is  to  be  implemented,  there  are  numerous 
practical  factors  to  be  considered  [5-8].  These  are  often  not  directly 
obvious  from  theoretical  discussions.  In  the  design  of  IR  imaging 
systems,  design  considerations  focus  mainly  on  the  detector  array. 
These  considerations  may  be  divided  into  three  basic  groups: 
mechanical,  electrical,  and  optical. 

Mechanical  considerations  include  secure  mounting  of  the  array  to 
the  imaging  system.  Movement  of  the  array  due  to  external  vibrations 
can  cause  loss  of  resolution  as  well  as  possible  damage  to  the  array. 
Any  moving  parts,  such  as  rotating  mirrors  should  be  sufficiently 
rugged  to  withstand  vibration  and  shocks  associated  with  airborne 
vehicles.  The  array  itself  should  also  be  rugged,  especially  in  the 
case  of  mosaic  arrays,  so  that  mechanical  and  electrical  connections  do 
not  fail.  A  further  consideration  may  be  the  actual  size  and  weight  of 
the  IR  imaging  system.  Excessive  size  and  weight  would  require 
possible  modification  of  existing  flight  vehicles.  Finally,  there 
should  be  provisions  for  cryogenic  systems  to  provide  cooling  for  the 


detector  arrays. 


Electrical  considerations,  much  like  the  mechanical 
considerations,  depend  on  their  physical  surroundings.  Ambient 
temperature  aay  have  an  influence  on  precision  electronics, 
particularly  for  high  altitude  flight. 

Optical  system  requirements  are  very  stringent  since  the  transfer 
of  IR  information  from  the  scene  to  the  detector  is  a  critical  step  in 
detection.  The  resolution  of  the  system  is  limited  by  the  detector 
spacing  of  the  focal  plane.  The  IR  optical  system  must  be  designed  so 
as  not  to  degrade  resolution.  The  design  of  IR  optical  systems  is 
further  complicated  by  the  limited  number  of  materials  transparent  to 


IR  radiation. 


3.0  Photogrammetr ic  Considerations 


The  integration  of  IR  imagery  into  Air  Force  and  DMA  operations 
involves  two  important  considerations.  These  are; 

1.  How  can  this  extra  capability  be  most  easily  added  with  the 
least  amount  of  modification  to  existing  equipment  and 
procedures? 

2.  In  the  absence  of  any  constraints  imposed  by  present  equipment 
and  procedures,  what  is  the  best  method  to  integrate  IR 
capabi 1 i ty? 

Stereocomparators  and  stereocompilers  are  computer  controlled 
image  measurement  systems  used  for  the  processing  of  visible  imagery. 
These  systems  are  multi-stage  stereocomparators  which  have  been 
interfaced  with  digital  computers  and  special  display  and  control 
equipment.  The  addition  of  the  computer  and  control  equipment  to  a 
stereocomparator  relieves  the  operator  of  many  of  the  laborious  and 
tedious  tasks  involved  with  the  processing  of  imagery  and  thus  greatly 
improves  operator  productivity.  From  a  systems  standpoint,  the 
stereocomparator  can  be  modeled  as  a  system  whose  inputs  are  Reseau 
data  and  images  stored  on  photographic  media.  The  outputs  of  the 
system  are  point  coordinates. 

In  order  to  consider  adding  IR  imagery  to  the  existing 
stereocomparator,  one  must  consider  how  to  input  the  IR  data.  It  is 
assumed  that  the  same  output  data  will  be  desired  for  either  IR  or 


3.1  Strip  Cameras 


A  strip  camera  operates  by  moving  film  past  a  narrow  slit  which 
functions  as  an  "open  shutter".  The  speed  of  the  film  is  synchronized 
to  the  ground  speed  of  the  aircraft  so  that  the  image  being 
photographed  remains  fixed  on  the  moving  film.  Advantages  of  strip 
photography  include  inherent  continuous  image  motion  compensation  and, 
since  it  has  no  shutter  mechanism,  camera  simplicity.  Disadvantages 
include  the  need  for  accurate  control  of  the  film  speed  and  poor  high 
altitude  performance.  -Strip  cameras  have  been  primarily  used  for  low 
altitude,  high  speed  reconnaissance  applications.  Since  strip 
photography  is  well  suited  for  linear  study  areas,  it  has  been  used 
extensively  in  commercial  applications  such  as  highway  and  railroad 
studies  and  selection  of  right  of  way  for  pipelines  and  power  lines 
[9]. 


3.2  Linear  TDI  Arrays 

The  basic  construction  of  a  linear  time-delay-and-integrate  (TDI) 
array  is  shown  in  Figure  7.  These  arrays  can  be  constructed  from 
either  CCD  or  CID  unit  cells.  Such  arrays  come  in  various  sizes  such 
as  1024  by  64  element  arrays  with  integration  steps  at  1,  4,  8,  16,  32, 
and  64  integrations.  These  arrays  are  operated  in  a  mode  analogous  to 
a  strip  camera.  The  relative  motion  between  the  aircraft  and  the 
ground  causes  the  image  to  move  down  the  array  from  line  to  line.  As 
the  image  moves,  the  charge  analog  of  the  image  is  electronically 
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Figure  7.  Linear  TDI  array. 


shifted  from  line  to  line.  Thus  when  the  image  reaches  the  last  line 
of  detectors,  the  charge  for  each  point  has  been  accumulated  by  several 
detectors.  This  integration  of  charge  increases  the  sensitivity  of  the 
imaging  device.  Note  that  moving  the  charge  analog  of  the  image  is 
analogous  to  moving  the  film  in  a  strip  camera.  In  the  strip  camera, 
the  exposure  is  controlled  by  varying  the  width  of  the  slit.  In  the 
TDI  array,  the  exposure  is  controlled  by  reading  out  the  signal  at 
different  lines  or  integration  steps.  That  is,  reading  out  the  image 
at  the  eighth  line  would  yield  a  signal  which  has  accumulated  charge 
for  a  shorter  period  of  time  than  reading  out  a  signal  at  the  32nd 
line.  Therefore,  the  brighter  the  scene,  the  fewer  number  of  lines  are 
needed  for  integration.  The  number  of  integrations  is  constrained  by 
the  size  of  the  array  and  also  by  the  amount  of  charge  a  unit  cell  can 
store.  "Overfilling"  a  unit  cell  can  cause  blooming  of  an  image. 

The  output  of  a  TDI  array  is  therefore  successive  lines  of  the 
scene.  It  is  then  necessary  to  assemble  these  lines  into  a  continuous 
picture.  This  procedure  is  complicated  by  the  fact  that  the  aircraft 
is  moving  and  since  each  strip  is  taken  at  a  different  time,  simply 
forming  a  photograph  by  assembling  many  lines  would  not  be  satisfactory 
for  precise  applications  such  as  mapping.  However,  if  the  flight 
dynamics  for  each  strip  are  known,  a  computer  can  process  the  strip 
data  and  assemble  them  into  frame  photographs.  Thus  it  is  possible 
that  data  from  strip  photography  can  be  made  useful  on  comparators 
designed  to  process  frame  photographs. 


3.3  Processing  of  Strip  Photography  Data 

Since  a  continuous  strip  photograph  is  usually  not  what  is  desired 
for  various  applications,  a  procedure  for  processing  strip  photography 
data  into  other  forms  such  as  frame  photographs  is  needed. 
Furthermore,  flight  path  nonuniformities  such  as  roll,  pitch,  yaw, 
altitude,  and  speed  cause  distortions  along  the  strip  photograph  which 
must  be  corrected. 

In  order  to  provide  a  means  for  performing  computational 
operations  for  strip  photography,  mathematical  procedures  have  been 
developed  [9-13] .  Case  [11]  suggested  the  use  of  Instantaneous 
Equivalent  Frame  Photos  (IEFP)  whereby  all  measured  points  are 
transferred  from  the  strip  to  a  ficticious  frame  photograph  having  the 
same  focal  length  but  a  single  instantaneous  position  and  attitude. 
The  selection  of  the  position  and  attitude  are  chosen  to  minimize  the 
deteriorating  effects  of  constantly  changing  position  and  attitude. 
Derenyi's  [ll]  method  of  line-to-line  orientation  is  a  similar  concept. 
In  this  method  the  strips  are  projected  down  to  the  ground.  After  the 
measured  strip  coordinates  have  been  projected,  the  resulting  ground 
points  are  projected  back  up  to  the  IEFP.  Thus  conventional  frame 
photographs  are  constructed  from  many  strip  photographs  with 
appropriate  corrections  applied. 

Three  analytic  systems  were  proposed  by  Derenyi  for  the  relative 
orientation  of  continuous-strip  imageries  produced  by  stereo  systems 
[13].  The  methods  he  described  are  applicable  to  continuous  strip 


camera*,  side  looking  radar*,  and  infrared  line-scanner  systems.  The 
first  of  these  proposed  methods  is  a  line-to~line  orientation  in  which 
two  lines  of  corresponding  points  are  oriented  relatively.  In  the 
second  method  small  sections,  rather  than  individual  lines,  are 
oriented  relatively.  Of  course,  it  must  be  assumed  that  any  changes 
which  occur  in  the  orientation  of  elements  within  these  sections  are 
negligible.  The  third  method  is  a  triple-channel  recording  scheme 
which  uses  a  third  channel  providing  a  vertical  view  of  the  terrain  in 
addition  to  the  two  regular  stereo  channels.  This  method  is  superior 
to  the  first  two,  however,  it  requires  a  more  complicated  sensor. 

3.3.1  Line-to-Line  Orientation 

In  a  strip  camera,  all  points  in  a  single  line  are  recorded 
simultaneously  and  thus  have  the  same  exterior  orientation.  It  is 
therefore  theoretically  possible  to  orient  two  lines  of  corresponding 
points  relatively  regardless  of  any  disturbing  effects  resulting  from 
continuously  changing  orientation  elements.  Typically,  the  coplanarity 
condition  is  applied  to  carry  out  the  relative  orientation.  Referring 
to  Figure  8,  the  points  on  line  BL  have  the  same  orientation  elements 
as  the  points  on  AR.  Thus  subsequent  line  pairs  can  be  oriented  to 
previous  ones  and  therefore  changes  in  orientation  elements  at  regular 
intervals  along  the  strip  can  be  determined. 

There  are,  however,  limitations  to  the  line-to-line  method.  Since 
neighboring  line  models  have  no  common  points,  transfer  of  the  scale 
from  line-to-line  along  the  flight  path  is  not  possible.  Also  only 
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three  of  the  five  independent  orientation  point*  can  be  found  since  the 
orientation  points  form  a  line  rather  than  a  plane.  The  other  two  must 
either  be  known  or  assumed  to  be  zero. 

Another  limitation  results  from  the  fact  that  the  look  angle  of 
the  terrain  points  across  the  flight  path  is  fixed.  Therefore,  terrain 
points  which  have  different  elevations  are  seen  at  different  times. 
Thus,  even  if  all  points  on  the  left  image  are  recorded  at  the  same 
instant,  the  recording  of  their  corresponding  points  on  the  right  image 
could  be  spread  over  an  interval  of  time.  One  result  of  this  is  that 
the  orientation  parameters  for  the  points  in  question  in  the  right 
image  could  be  nonuniform.  Another  negative  result  is  that  the  base 
length  from  point  pair  to  point  pair  is  a  function  of  the  above 
mentioned  time  interval. 

3.3.2  Section-to-Section  Orientation 

In  this  method  it  is  assumed  that  changes  which  occur  in  the 
orientation  elements  within  a  section  of  the  strip  are  negligible. 
Therefore,  rather  than  taking  a  line  as  a  unit,  a  small  section  of  the 
strip  is  taken  as  a  unit  as  shown  in  Figure  9.  This  method  is  similar 
to  the  line-to-line  method. 

The  main  limitation  of  the  sec t i on-t o-sec t i on  method  is  that  the 
orientation  of  a  sensor  does  not  actually  remain  constant,  as  was 
assumed,  during  the  recording  of  a  section.  Derenyi  [13]  conducted  a 
comprehensive  study  of  the  effect  of  the  neglected  changes  in  the 
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Figure  9a.  Storage  of  section  data  for  section-to-section  orientation  (13). 


Figure  9b.  Coordinate  system. 


orientation  on  the  solution.  The  direction  of  flight  is  X,  the 
transverse  coordinate  is  Y,  and  the  altitude  is  Z.  Let  u,<|>  ,  and  k  be 
the  rotation  about  the  X,  Y,  and  Z  axes  respectively  as  shown  in  Figure 
9b.  It  was  found  that  small  changes  in  dbz  and  dby  can  distort  the 
solution  of  d<}>  and  die .  These  are  the  two  most  critical  elements.  The 
solution  for  du  ,  dby,  and  dbz  were  approximately  their  average  within 
a  particular  section.  Lastly,  it  was  found  that  the  effect  of 
nonuni formity  of  the  orientation  is  inversely  proportional  to  the 
length  of  a  section.  A  longer  section,  however,  weakens  the  validity 
of  the  constant  orientation  assumption  since  the  recording  time  of  the 
section  is  longer. 


3.3.3  Triple  Channel  Recording 


Derenyi  suggested  adding  a  third  channel  which  views  the  terrain 
vertically  downward.  Thus  the  three  channels  will  be  forward, 
vertical,  and  aft.  All  three  images  would  be  recorded  side  by  side  as 
shown  in  Figure  10.  Points  which  are  recorded  simultaneously  by  the 
three  channels  are  equivalent  to  three  lines  on  a  frame  photograph. 
Lines  A  ,  B  ,  and  C  are  equivalent  to  three  lines  on  a  frame 
photograph  taken  from  the  same  space  position.  Lines  A  and  C 
correspond  to  similar  lines  along  the  left  and  right  edges  of  a  frame 
photograph,  and  line  B  corresponds  to  a  line  passing  through  the 
principal  point.  With  this  scheme,  a  complete  relative  orientation, 
strip  triangulation,  and  transfer  of  scale  along  the  flight  path  are 
possible.  Of  course,  considerable  complexity  is  added  to  the  sensor. 


3.3.4  Processing  Stereo  Line  Image  Data 


Recent  work  has  been  performed  by  Gibson  [14]  at  the  Canada  Centre 
for  Remote  Sensing  applying  the  procedures  described  above.  The 
objective  of  this  work  is  to  establish  the  feasibility  of  using  line 
scanning  image  sensors  to  produce  imagery  suitable  for  map  making.  The 
line  imaging  sensors  used  were  visible/near  infrared  devices,  however, 
the  results  are  applicable  to  line  scanning  image  sensors  in  general. 

Gibson  has  recently  studied  two  modes  of  stereo  imagery.  The 
first  one  used  a  Daedalus  Multi-Spectral  Scanner  (MSS)  which  acquired 
data  by  flying  parallel  paths  with  601  sidelap  of  the  lines.  The  other 
set  of  data  was  obtained  using  the  CCRS  Multi-Element-Imaging-System 
(MEIS)  II  line  imager.  Two  externally  mounted  mirrors  provided  both 
forward-looking  and  aft-looking  channels.  A  third  channel  looking 
straight  down  was  also  used  allowing  the  triple  channel  method 
described  earlier  to  be  investigated.  An  inertial  navigation  system 
(INS)  was  used  to  simultaneously  record  the  flight  dynamics  of  the 
aircraft.  For  the  MSS  data,  a  high  altitude  laser  profiler  was  us“d  to 
accurately  record  the  altitude  of  the  aircraft. 

The  data  studied  in  this  project  was  obtained  from  a  five  minute 
flight  line  at  8200  meters  above  ground  level.  The  flight  data  which 
was  analyzed  consisted  of  pitch,  heading,  altitude,  and  velocity.  Roll 
data  was  not  used  since  both  imaging  systems  had  roll  correction 
capabi 1 i ty . 
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The  imaging  system  had  an  instantaneous  field  of  view  per  pixel  of 
.7  milliradians  resulting  in  a  pixel  size  of  5.7  meters.  The  following 
is  a  summary  of  the  pixel  displacements  resulting  from  aircraft  motion 
on  a  typical  mission  [ 1  A]  . 

Pitch  angle  variations  had  two  observed  components.  One, 
resulting  from  the  autopilot,  was  of  large  amplitude  and  low  frequency. 
The  other  was  a  small  amplitude,  high  frequency  vibration  induced 
component.  The  pixel  displacements  resulting  from  pitch  angle 
displacements  ranged  from  1.3  to  16.2  pixels  with  a  maximum  change  over 
a  20  second  interval  of  10.0  pixels. 

Heading  angle  variations  also  were  composed  of  both  high  and  low 
frequency  components.  The  effects  of  heading  angle  variations  were 
calculated  in  terms  of  the  relative  shift  of  one  end  of  the  line  with 
respect  to  the  other.  The  pixel  displacements  ranged  from  1.8  to  23.2 
pixels  with  a  maximum  change  in  20  seconds  of  1 A . 3  pixels. 

Altitude  changes  caused  a  low  frequency  effect  which  highly 
correlates  with  pitch  angle  changes.  The  result  of  altitude  changes  is 
the  apparent  stretching  or  shrinking  of  the  line  of  imagery  on  the 
ground.  The  resulting  pixel  displacement  had  a  maximum  of  5.1  pixels 
and  a  maximum  change  in  20  seconds  of  1.1  pixels. 

Velocity  changes  occur  both  along  and  across  the  flight  track. 
Along  track  variations  cause  stretching  or  shrinking  of  the  spacing  of 
the  image  lines  on  the  ground.  The  maximum  pixel  displacement  caused 
by  along  track  velocity  variations  was  13.1  pixels.  Across  track 


variations  cause  a  side  or  transverse  shift  of  a  line  of  imagery  with 
respect  to  its  nearest  neighbors.  The  maximum  pixel  displacement 
resulting  from  this  was  5.0  pixels.  As  can  be  seen  from  these  data, 
very  accurate  flight  dynamics  data  are  needed  to  produce  mapping 
quality  images. 

In  processing  the  stereo  line  imager  data,  Gibson  used  techniques 
similar  to  those  employed  for  conventional  photography.  That  is, 
relative  and  absolute  orientation  parameters  were  obtained,  however, 
the  processing  was  complicated  somewhat  because  of  the  need  to  remove 
low  frequency  errors  of  the  inertial  navigation  system  data.  The 
coplanarity  condition  was  used  to  relate  the  stereo  images  using  ground 
points  whose  co-ordinates  were  not  known.  The  collinearity  condition 
was  used  for  relating  the  imagery  to  known  ground  control  points.  Both 
the  coplanarity  and  collinearity  equations  were  linearized  using  the 
first  term  of  a  Taylor  Series  expansion.  The  resulting  set  of 
equations  was  used  to  calculate  the  short  term  offset  and  drift  errors 
of  the  inertial  navigation  system  data.  Position  and  altitude  data 
were  then  used  for  the  geometric  correction  of  the  imagery  data.  The 
details  of  the  coplanarity  and  collinearity  condition  equations  and  the 
geometric  correction  process  can  be  found  in  a  previous  publication  by 
Gibson  [15] . 

In  order  to  be  able  to  compute  run  times,  the  number  of  operations 
per  pixel  and  the  number  of  operations  per  second  of  the  computer  must 
be  known.  Gibson,  for  example,  used  1024  pixels  per  line  and  to 


accomplish  the  necessary  corrections  to  the  raw  data  approximately 
75,000  operations  per  line  were  required.  Modern  minicomputers  are 
capable  of  1  to  5  million  floating  point  operations  (MFLOPS)  per 
second.  These  co'mputers,  therefore,  would  require  approximately  15  to 
75  milliseconds  of  computational  time  to  process  each  line  of  the 
image.  With  modern  microprocessors  and  optimized  algorithms,  a 
dedicated  machine  operating  at  5  to  10  MFLOPS  should  be  attainable.  If 
custom  Very-High-Speed-Integrated-Circuit  (VHSIC)  technology  is  used, 
speeds  of  up  to  100  MFLOPS  could  be  realized.  Thus  execution  times  of 
approximately  750  microseconds  per  line  are  theoretically  possible. 
With  such  a  dedicated  processor,  a  1024  line  image  could  be  processed 
in  less  than  a  second. 

Note,  however,  these  run  time  estimates  are  for  processing  the  raw 
data  and  applying  corrections  for  flight  perturbations.  Run  time 
estimates  for  operations  such  as  image  translation,  rotation, 
magnification,  etc.,  are  discussed  in  the  Appendix. 


4.0  Resolution  Model 


In  the  past,  several  methods  have  been  proposed  for  the  evaluation 
of  swept  linear  array  system  performance.  Trade-offs  between 
resolution  and  detector  signal  to  noise  ratio  (SNR)  have  determined 
optimal  detector  size  as  a  function  of  optical  modulation  transfer 
function  (MTF) ,  detector  physical  characteristics,  aperture,  and 
contrast  [16] .  The  effects  of  detector  shape  (rectangular  or 
elliptical)  and  detector  integration  interval  on  system  performance 
have  also  been  modeled  [17].  Angular  uncertainty  in  swept  linear 
detector  arrays  has  been  estimated  for  specific  point  tracking  and 
aerial  reconnaissance  systems  [  18—20] •  In  most  cases,  these  models 
apply  only  to  specific  applications  because  of  simplifying  assumptions 
made  about  detector  geometry,  blur  circle,  and  other  system  parameters. 

The  system  resolution  model  described  in  this  section  assumes 
fewer  design  constraints.  Recognizing  the  conflicting  requirements  of 
high  spatial  resolution  (small  detector  elements)  and  high 
signal-to-noise  ratio  (large  detector  area),  the  object  of  this 
analysis  approach  is  to  quantify  the  trade-offs  between  optical 
performance,  platform  vibration,  flight  dynamics,  resolution,  and 
detector  geometry  for  a  swept  linear  array  imaging  system.  The 
scenario  shown  in  Figure  11  assumes  a  linear  time-delay-and-integrate 
(TDI)  array  used  in  a  push-broom  scan  mode.  This  model,  however,  is 
general  enough  to  accommodate  staring  arrays  or  mechanically  swept 
linear  arrays  with  interleaved  detector  elements. 


4.1  Resolution  of  Solid  State  laagers 


Traditional  sampling  theory  assumes  an  ideal  spatial  (two 
dimensional)  sampling  function 
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that  is  composed  of  an  infinite  array  of  Dirac  delta  functions  [21] . 
Given  a  spatially  bandlimited  image,  the  sampling  period  must  be  equal 
to  or  greater  than  one-half  the  period  of  the  finest  detail  (or  spatial 
frequency)  in  the  image  if  it  is  desired  to  reconstruct  a  sampled  image 
having  the  same  detail.  This  is,  of  course,  equivalent  to  the  well 
known  sampling  theorem  of  Shannon.  The  spacing  of  Dirac  delta 
functions  is  given  by  the  Nyquist  criterion 
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where  wx'wy  are  image  cutoff  frequencies.  The  importance  of  two- 
dimensional  sampling  theory  is  that  it  sets  a  theoretical  limit  on  the 
spatial  frequencies  that  may  be  reconstructed  from  sampled  images. 

Practical  imaging  focal  plane  arrays  or  linear  scanned  arrays 
cannot  achieve  the  performance  of  ideal  sampling  functions.  The 
consequence  of  using  sampling  pulses  of  finite  width  instead  of  Dirac 
delta  functions  is  that  the  image  may  be  undersampl ed.  Since  the 
signal-to-noise  ratio  of  a  detector  element  is  related  to  the  amount  of 
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incident  quantum  energy  integrated  in  time  and  space,  designers  try  to 
maximize  the  detector  responsive  area. 

Staggered  arrays  and  TDI  arrays  are  two  approaches  used  to 
maximize  detector  area  under  a  centerline  spacing  constraint.  The 
ratio  of  detector  centerline  spacing  to  detector  width  is  called  the 
fill  factor.  Fill-factors  less  than  unity  may  be  due  to  geometric 
design  constraints.  Fill-factors  greater  than  unity  are  possible  in 
linear  scanned  arrays  having  even  and  odd  detectors  in  two  staggered 
rows  as  shown  in  Figure  12. 

Practical  imaging  systems  not  only  sample  with  finite  width 
pulses,  but  they  sample  moving  images  over  some  finite  time  period. 
Image  motion  relative  to  the  focal  plane  causes  a  blurring  effect. 
This  effect  contributes  to  reduced  resolution  of  the  reconstructed 
image  in  addition  to  sampling  pulse  (spatial)  effects.  Image  motion 
might  result  from  motion  of  the  image  itself  (moving  tanks , ships) ,  from 
vibration  of  the  sensor  platform  (reconnaissance  aircraft),  and  from 
the  mechanical  scanning  of  the  detector  element  or  array.  Cell 
crossing  velocity  refers  to  the  velocity  of  a  blur  spot  of  a  point 
source  as  it  crosses  a  detector  element. 

Optical  system  designers  often  use  an  arbitrary  but  useful  measure 
of  resolution,  the  Sparrow  criterion,  as  an  indication  of  limiting 
performance  for  a  given  size  system  [22].  According  to  this  criterion, 
two  adjacent  point  sources  (not  necessarily  of  equal  intensity)  are 
considered  resolved  if  their  combined  diffraction  pattern  on  the  focal 
plane  has  no  maximum  between  the  two  point-images. 


A  definition  of  optimal  resolution  for  solid-state  imaging  systems 
is  presented  in  this  section  that  combines  the  concepts  of  sampling 
theory  and  the  Sparrow  criterion.  With  such  a  model,  a  systers 
analysis  can  be  performed  to  calculate  the  resolution  of  a  given 
system.  Such  factors  as  detector  geometry,  optical  components, 
electronic  timing,  and  virtual  image  motion  can  be  considered.  The 
model  is  novel  in  that  it  combines  spec i f i ca t i ons  of  the  optical  train, 
stabilized  platform,  detector  a  ray,  flight  dynamics,  and  electronics 
into  one  model . 


By  the  Nyquist  criterion,  the  smallest  resolvable  detail  is  given 
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where  d  is  the  centerline  spacing  between  detector  elements.  Depending 
on  the  width  of  the  point  spread  function,  resolution  can  be  much  worse 
than  this  but  it  can  never  be  any  better.  A  definition  of  resolution 
for  two  point  sources  on  a  detector  array  must  be  modified  from  a 
continuous  Sparrow  criterion  since  resolution  must  be  an  integer 
multiple  of  d.  Let  the  discrete  Sparrow  criterion  be  defined  such  that 
two  adjacent  point  sources  are  resolved  if  the  energy  of  their  combined 
sampled  point  spread  functions  do  not  have  a  unique  maximum.  Figure 
13a  illustrates  the  quantum  energy  from  a  single  point  spread  function 
integrated  by  detector  elements.  Figure  13b  and  13c  show  two  sets  of 
point  spread  functions  spaced  at  a  distance  2d  apart.  The  sample 
period  is  d  in  both  cases.  In  Figure  13b  the  two  points  are  resolved. 


In  Figure  13c,  however. 
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the  energy  of  the  combined  sampled  point  spread  function  has  a  single 
maximum  and  thus  the  two  points  are  not  resolved. 


The  discrete  Sparrow  criterion  may  be  expressed  very  simply  if  we 
consider  only  noiseless  optimal  resolution  of  two  equal  energy  point 
sources.  Assume  a  point  spread  distribution  centered  on  a  detector 
element,  as  shown  in  Figure  13a.  Denote  the  quantum  energy  incident  on 
that  detector  over  some  time  interval  as  Qo.  Let  Q1 ,  Q2,  Qn  represent 
the  energy  from  the  same  point  source  falling  on  adjacent  detector 
elements.  Two  equal  energy  point  sources  are  resolved  at  a  minimum 
distance  of  2d  on  the  focal  plane  if 

0o  >  20,  -  02  (4) 

The  point  sources  are  assumed  to  be  noiseless  and  contributions  to  Qi 
from  other  sources  are  neglected. 

The  decision  to  define  resolution  in  terms  of  point  sources  is 
arbitrary.  Though  it  is  one  of  many  ways  to  characterize  system 
resolution,  it  facilitates  the  calculation  of  constraints  necessary  to 
achieve  Nyquist  sampling. 

Resolution  is  not  the  same  thing  as  radiometric  accuracy.  The 
resolution  of  a  solid  state  sensor  system  is  ultimately  limited  by  the 
spatial  sampling  rate  or  detector  spacing.  However,  sub-pixel 
accuracies  are  possible  whenever  correlation  in  time  or  space  exists. 
Automated  edge  location  measurements  of  images  have  achieved  accuracies 
of  better  than  0.30  pixels  in  the  presence  of  10X  RMS  noise  [23]. 


Greater  accuracies  have  been  demonstrated  by  applying  Kalman  filtering 
techniques  to  the  tracking  of  moving  point  sources. 

Using  a  resolution  criteria  based  on  point  sources,  a  point  spread 
function  is  derived  which  describes  light  intensity  incident  on  the 
focal  plane.  The  shape  and  width  of  a  point  spread  function  depends  on 
optics,  image  motion  (due  to  object  or  camera  motion)  and  atmospheric 
effects.  Careful  optical  design  can  minimize  image  degradation  due  to 
diffraction  limiting  and  optical  aberration  effects. 

Blur  resulting  from  several  sources  is  conservatively  estimated  by 
summing  blurs  resulting  from  individual  aberrations  and  diffraction 
limiting  effects  [24].  Individual  blur  spot  sizes  may  be  estimated 
from  lens  type,  focal  length,  and  aperture.  Platform  vibration 
occuring  during  a  charge  integration  period,  has  the  multiplicative 
effect  of  spreading  the  blur.  Assuming,  as  does  [20] ,  that  the 
resulting  point  spread  function  is  Gaussian  distributed,  the  Gaussian 
shaped  profiles  are  easily  convolved. 
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The  resulting  measure  of  image  blur  is  easily  translated  into  a  point 
spread  function. 
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Generation  of  the  point  spread  function  from  optical  and  vibration 
parameters  is  shown  schematically  in  Figure  14. 


4.2  System  Model 


An  aerial  reconnaissance  scenario,  illustrated  in  Figure  11, 
assumes  a  linear  t ime-delay-and-integrate  (TDI)  array  used  in  a 
pushbroom  scan  mode.  This  model,  however,  is  general  enough  to 
simulate  staring  arrays,  as  well  as  mechanically  scanned  linear  arrays 
with  interleaved  (fill  factor  greater  than  unity)  detector  elements. 

Whereas  point  spread  functions  depend  on  optical  performance  and 
platform  vibration,  system  resolution  depends  on  three  additional 
parameters:  flight  dynamics,  detector  geometry,  and  TDI  timing.  The 
quantum  charge  accumulated  under  each  detector  element  is  determined  by 
integrating  the  moving  point  spread  function  with  respect  to  these 
three  parameters.  It  is  possible  to  determine  conditions  which 
maximize  detector  responsive  area  for  a  Nyquist  sampling  rate  by 
applying  the  Discrete  Sparrow  Criterion  to  the  discrete  integrated 
charge  function. 

As  can  be  seen  in  Figu:  11,  flight  dynamics  affect  the  cell 
crossing  velocity  or,  equivalently,  the  velocity  which  a  point  spread 
function  moves  across  a  detector  cell  or  element.  Since  level, 
straight,  and  constant  velocity  flight  is  assumed  in  this  simulation, 
cell  crossing  velocity  is 

V  *f 


where  Vg  is  the  sensor  velocity  with  respect  to  ground,  h  is  the 
altitude  of  the  sensor  and  f  is  the  focal  length  of  the  sensor  optics. 
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Detector  centerline  spacing,  d,  and  fni  far’cr 
geometry  parameters  modeled  in  this  simulation.  As  prev.ously 
discussed,  a  smaller  fill~factor  move  closely  approximates  a  Dirac 
delta  function  at  the  cost  of  detector  s 1 gna 1 - t c~ no i s e  ratio. 

A  unique  function  of  this  model  is  that  the  TDI  charge  integration 
period,  Tj  ,  is  not  necessarily  equal  to  T  ,  the  time  required  for 
a  blur  spot  to  travel  from  one  detector  element  to  another.  In  this 
model  it  is  assumed  that 


In  other  words  it  is  not  assumed  that  a  detector  is  always  "on"  and 
thus  accumulating  signal.  Consider,  for  example,  a  C ID  element  which 
is  turned  "off".  During  the  time  the  element  is  off,  charge  carriers 
which  are  generated  by  incident  radiation  are  not  accumulated  but  are 
injected  into  the  substrate  and  lost.  This  model  has  the  flexibility 
to  handle  such  a  situation. 

Radiant  energy  originating  from  a  point  source  and  incident  on  a 
detector  element  is  found  by  integrating  flux  density,  described  by  a 
point  spread  function  S(x),  incident  at  the  detector  during  one 
integration  period.  Quantum  flux  (for  a  one_dimens i onal  case)  is  given 
by 

♦  (k)  -  /  X+/2  S(x,d*  (9 
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where  S(x)  is  a  Gaussian  point  spread  function.  Assuming  a  constant 


ground  velocity,  we  have 
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Integrating  (10)  over  one  integration  period  results  in  the  incident 
energy  on  the  i  —  t h  detector  elements. 
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Combining  (11)  and  (12)  and  solving  for  an  equality  results  in  a  limit 
for  the  system  parameters  meeting  the  optimal  resolution  criteria. 


A. 3  Application  of  the  System  Model 

The  resolution  model  presented  in  this  section  provides  a 
framework  for  the  analysis  of  imaging  systems  using  discrete  arrays. 
This  model  can  be  applied  for  systems  employing  staring  array  sensors 
as  well  as  electronically  or  mechanically  scanned  linear  arrays.  By 
varying  system  parameters,  it  is  possible  to  determine  the  best 
trade-offs  between  these  system  parameters  and  system  performance. 
System  parameters  included  in  the  resolution  model  fall  into  three 
categories.  These  are  optical  system  parameters,  focal  plane 
parameters,  and  flight  dynamics  parameters. 


The  optical  system  parameters  include  aperture  size,  focal  length, 
abberations,  astigmatism,  and  other  influences  on  the  point  spread 
function.  These  are  constrained  for  the  most  part  by  the 
state-of-the-art  in  optical  lens  manufacture. 

The  sensor  or  focal  plane  parameters  include  detector  geometries, 
fill  factors,  TDI  charge  integration  period,  and  centerline  spacing. 
The  system  designer  must  determine  the  optical  trade-offs  with  respect 
to  these  parameters  while  keeping  in  mind  the  importance  of  considering 
noise  sources. 

Flight  dynamics  parameters  include  sensor  velocity,  altitude,  and 
platform  vibration.  These  parameters  are  constrained  largely  by  the 
available  airborne  platform  and  the  particular  mission  scenario. 

By  modeling  a  system  using  the  resolution  model,  one  must  assume 
certain  parameter  values  consistent  with  the  mission  scenario  and 
system  speci f i cat i ons .  By  varying  the  values  of  focal  plane 
parameters,  it  is  possible  to  determine  the  best  system  trade-offs  and 


optical  parameter  values. 


5.0  Conclusions  snd  Recommendations 

A  survey  of  existing  IR  imaging  systems  has  been  performed.  The 
most  promising  systems  for  use  for  aerial  photography  are  scanned 
linear  arrays  of  CCD  or  CID  unit  cells  operated  in  a  time-delay- 
and-integrate  (TDI)  mode.  These  types  of  imaging  systems  are  basically 
an  electronic  strip  camera.  Imagery  produced  by  these  systems,  after 
suitable  data  processing,  can  be  used  for  mapping  and  targeting 
applications .  Gibson  reported  maximum  pixel  di splacements  during  a  20 
second  interval  for  uncorrected  imagery  to  be  10.0  pixels  for  pitch 
angle  variations,  14.3  pixels  for  heading  angle  variations,  1.1  pixels 
for  altitude  changes,  and  5.0  pixels  for  velocity  changes.  From  these 
data  it  is  obvious  that  uncorrected  imagery  is  not  accurate  enough  for 
most  mapping  and  targeting  applications.  When  high  accuracy  is 
required,  three  dimensional  flight  data  such  as  that  from  an  inertial 
navigation  system,  should  be  simultaneously  recorded  with  the  imagery. 
It  is  theoretically  possible  to  perform  corrections  on  existing  imagery 
using  knov-n  ground  points  rather  than  flight  data.  Such  a  process, 
however,  is  tedious  and  dependent  upon  the  imagery  itself.  Correction 
of  such  imagery,  therefore,  is  not  amenable  to  developing  generalized 
software  and  would  be  limited  to  very  special  applications. 

A  system  model  useful  for  calculating  the  resolution  of  an  imaging 
system  was  also  developed.  This  model,  which  is  adaptable  to  linear 
swept  arrays  as  well  as  two-dimensional  staring  arrays,  can  be  used  to 
calculate  or  to  perform  system  design  trade-offs  under  a  system 
resolution  constraint. 


48 


It  is  recommended  that  future  systems  go  to  an  all  digital  format 
in  order  to  capitalize  on  the  advances  in  computer  image  processing  and 


computer  graphics.  Such 

systems 

could 

store 

both 

visible  and  IR 

imagery  in  the  same  format 

allowing 

the 

use  of 

the 

same 

computer 

processing.  Both  visible  and 

IR 

imagery 

could 

be 

recorded 

simultaneously  along  with  the  flight  data.  After  processing  it  would 
then  be  possible  to  electronically  display  either  or  overlay  both 
images.  Other  advantages  of  an  all  electronic  digital  system  include 
the  increased  dynamic  range  of  electronic  systems  over  film,  the 
elimination  of  expensive  and  restrictive  mechanical  systems  for 
plotting  and  display,  accurate  and  fast  x  and  y  coordinate  computation 
since  each  pixel  is  digitally  stored,  the  elimination  of  film  induced 
distortions,  and  the  ability  to  scroll  through  a  flight  path  on  a  CRT 
essentially  as  though  you  were  looking  down  from  the  aircraft  [14,15). 
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Appendix  -  Real  Time  Response 


A  requirement  exists  for  an  analytical  stereoplotter  featuring  the 
realtime  response  of  digital  stereomodels  to  handwheel  and  footwheel 
movements.  In  order  to  establish  the  feasibility  and  design  of  such  a 
system,  a  definition  of  "real-time"  is  needed.  The  following  section 
defines  the  term  in  the  context  of  this  project,  recommends  a  computer 
response  time,  and  discusses  the  trade-offs  in  the  design  of  such  a 
real-time  system. 

In  a  real-time  digital  analytical  stereoplotter  (DAS)  or 
stereocomparator,  there  should  be  no  perceptible  time  lag  between  the 
handwheel  and  footwheel  movements  and  the  movement  of  the  floating 
mark.  To  meet  this  requirement,  the  response  time  of  the  image 
processing  hardware  should  be  less  than  the  human  response  time  visual 
stimulus.  Since  re^l-time  DAS  response  places  such  great  demands  on 
the  image  processing  equipment,  a  trade-off  exists  between  what  is  an 
acceptable  DAS  response  time  and  hardware  complexity,  cost  and  risk 
(feasibility) . 

Reaction  time  varies  among  humans  as  well  as  with  the  sensor  used 
[25] .  The  quickest  reaction  time  is  in  response  to  a  sound  and  is 
about  150  mi 1 1 iseconds .  Response  to  a  tactile  sensation  of  pain  may 
take  up  to  700  milliseconds.  Under  ideal  circumstances,  a  "fast" 
person  can  react  to  a  visual  stimulus  in  200  mi  1 1 iseconds .  This  does 
not  imply  that  a  computer  response  time  of  190  milliseconds  will  go 
unnoticed,  but  that  it  will  result  in  a  combined  system  response  time 


of  390  Billiseconds. 


Computer  ergonomic  studies  have  provided  recommendations  for 
maximum  acceptable  computer  response  time  for  various  tasks  [25-27] . 
The  recommended  response  time  measured  from  the  act  of  drawing  with  a 
light  pen  to  a  visual  response  was  100  milliseconds.  Longer  response 
times  resulted  in  a  noticeable  lag  and  loss  in  productivity.  The  same 
response  time  was  reported  for  acoustic  feedback,  such  as  a  "click",  in 
response  to  a  key  depression. 

Manipulating  computer  imagery  in  real  time  can  be  a  difficult  and 
computation  intensive  operation.  As  a  simple  example,  consider  an 
image  of  512  pixels  per  line  by  512  pixels  per  line  with  eight  bits  per 
pixel.  This  corresponds  to  a  moderately  high  resolution  video  image. 
The  generalized  linear  transformation  or  affine  transformation 
represents  axonometric  distortions  where  parallel  straight  lines  map 
into  parallel  straight  lines.  This  algorithm  is  given  by: 


(A1 

*  b,  j  t  b2  k 
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where  (p,q)  represent  the  transformed  pixel  coordinates  and  (j,k) 
represent  the  original  pixel  coordinates.  The  affine  transform  is  used 
for  image  translation,  rotation,  and  scale  change.  To  generate  a  new 
set  of  coordinates  for  each  pixel  in  this  example  requires  1,048,576 
additions  and  1,048,576  multiplications.  Since  the  new  coordinates  may 
not  fit  the  grid  format  of  the  video  raster  display,  a  resampling 
function  is  necessary  to  assign  values  to  all  new  pixel  locations.  The 
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